Many modern vehicles include advanced driver assistance systems (ADAS) to provide automated safety and/or assisted driving functionality. Many next generation vehicles will likely include autonomous driving (AD) systems to control and navigate the vehicles independent of human interaction. Certain ADAS and AD systems perform driving operations based on a situational awareness of the vehicle's surroundings and the state of the vehicle. The situational awareness of the vehicle can include a classification, such as a situational annotation, of the current driving situation in which the vehicle is operating. Machine learning techniques are one of the possibilities used to classify a situation and generate a situational annotation. In existing systems, a situation is classified based on data from a single source of data (such as a single sensor modality) and/or object-level data from multiple sensors. Classifying a situation using these and other existing approaches, however, can be computationally inefficient and inaccurate.